Systems and methods for coordinating radar data and image data to track a flight of a projectile

ABSTRACT

A system for coordinating radar data and image data to track a flight of a projectile includes a radar, an imager, and a controller. The imager provides an image of an area into which a projectile is to be launched. The controller receives the image and an identification of a target within the image and identifies a target line connecting the target and a launch position of the projectile. The controller also receives data from the radar relative to at least a portion of a trajectory of the projectile, generates image based trajectory data by correlating the radar data and the image, and alters the image to include the image based trajectory data within the image and the target line within the image.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a Continuation application of U.S. patentapplication Ser. No. 15/050,456 filed Feb. 22, 2016, now U.S. Pat. No.10,052,542; which is a Continuation application of U.S. patentapplication Ser. No. 13/138,167 filed Aug. 9, 2011; which is a 371application of PCT Application Serial No. PCT/EP2010/051102 filed Jan.29, 2010; which claims priority benefit to U.S. Provisional PatentApplication Ser. No. 61/202,111 filed Jan. 29, 2009. In addition, PCTApplication Serial No. PCT/EP2010/051102 is a Continuation-in-Partapplication of U.S. patent application Ser. No. 11/630,904 filed Jul.20, 2009, now U.S. Pat. No. 8,085,188; which is a 371 application of PCTApplication Serial No. PCT/DK2005/00456 filed Jul. 1, 2005; which claimspriority benefit to U.S. Provisional Patent Application Ser. No.60/584,483 filed Jul. 2, 2004. Specifications of the aboveapplications/patents are expressly incorporated herein, in theirentirety, by reference.

PCT Application No. PCT/EP2010/051102 is also a continuation-in-part ofU.S. patent application Ser. No. 11/630,904, filed 20 Jul. 2009, nowU.S. Pat. No. 8,085,188, which is a National Stage of PCT ApplicationNo. PCT/DK05/00456, filed 1 Jul. 2005, which is a nonprovisional of U.S.Provisional Application No. 60/584,483, filed 2 Jul. 2004.

SUMMARY

The present disclosure relates to a system comprising a camera and aradar, for example, for combining imaging and motion determination ofathletes' movements and/or for coordinating radar data and image data totrack a flight of a projectile.

According to aspects of the disclosure, a system for coordinating radardata and image data to track a flight of a projectile includes a radar,an imager, and a controller. The imager provides an image of an areainto which a projectile is to be launched. The controller receives theimage and an identification of a target within the image and identifiesa target line connecting the target and a launch position of theprojectile. The controller also receives data from the radar relative toat least a portion of a trajectory of the projectile, generates imagebased trajectory data by correlating the radar data and the image, andalters the image to include the image based trajectory data within theimage and the target line within the image.

In accordance with aspects of the disclosure, a device for coordinatingradar data and image data to track the flight of a projectile comprisesa controller receiving an image and an identification of a target withinthe image and identifying a target line connecting the target and alaunch position of the projectile. The controller also receives datafrom a radar relative to at least a portion of a trajectory of theprojectile, generates image based trajectory data by correlating theradar data and the image, and alters the image to include the imagebased trajectory data within the image and the target line within theimage.

According to aspects of the disclosure, a method for coordinating radardata and image data to track a flight of a projectile comprisesreceiving, by a controller, an image and an identification of a targetwithin the image as well as radar data relating to at least a portion ofa trajectory of a projectile traveling through a field of viewrepresented by the image; identifying, by the controller, a target lineconnecting the target and a launch position of the projectile;generating, by the controller, image based trajectory data bycorrelating the radar data and the image; and altering the image toinclude the image based trajectory data and the target line within theimage.

In one exemplary aspect, the invention relates to an assembly includinga radar and an imaging device. The assembly further includes acontroller adapted to: receive radar data from the radar and an imagefrom the imaging device; determine, from the radar data, data of objectsimaged by the imaging device, the data being associated with one ofpositions, directions, trajectories, or planes of movement of theobjects; and identify, from the data, a position of impact of a firstobject of the objects and a second object of the objects at a correctposition in the image by providing movement data. The movement dataincludes, for the first object, a spin after the impact, and for thesecond object, a 3D impact vector at the impact.

In the present context, a radar is any element adapted to outputelectromagnetic radiation, such as radio waves, visible or invisibleradiation, and from such radiation reflected from an element, determineparameters related to the elements position, velocity, spin,acceleration, movement or the like. Preferably, the radar operates onthe basis of radio waves and may be a CW Doppler radar.

Also, an imaging device may be any type of element adapted to generatean image of e.g. a scene or object. This device may be a camera, such asa still image camera or a video camera. Preferably, the device isadapted to have a variable zoom or enlargement of elements in the image,and different parameters of the image/video generation preferably areselectable, such as shutter time and/or number of frames per second.

The controlling means are adapted to receive an image from the imagingdevice and adapt the image by providing in the image data relating tothe radar data, the altering of the image, such as altering the image ofelements imaged on the basis of the radar data.

In one embodiment, the controlling means is adapted to identify, fromthe radar data and/or the image, a position of impact of two of theobjects and illustrate:

-   -   for a first of the objects at least one of: a trajectory, a 3D        launch vector and spin after impact and    -   for a second of the objects at least one of trajectory,        direction/angle of movement and 3D impact vector at impact.

Preferably, the controlling means is adapted to provide informationrelating to an impact between a golf club and a golf ball, both presentin the fields of view of the radar and imaging device, the golf ballbeing the first of the objects and the golf club being the second of theobjects.

In one situation, the controlling means is adapted to determine andillustrate one or more of:

-   -   a face angle of the club at impact,    -   a relation between the ground and an axis of a club head of the        club at impact, and    -   a point of impact of the club head.

In the present context, the face angle may be an angle of a, usuallyplane or at least reasonably plane, surface or face of the club used forlaunching the ball.

Also, the relation between the ground and the axis may be an anglebetween the ground at the position of launch (horizontal or not) and alower surface or a mean axis of the club head.

The point of impact of the club head may be a position of a launchingsurface of the head or a position within a circumference of the clubhead in a plane perpendicular to a direction of movement of the clubhead at impact.

In general, the position, trajectory or plane may be illustrated in theimage at the correct position, which may be obtained by a calibrationbetween the radar and the imaging device.

In the situation of a golfer swinging a club and launching a golf ball,the tee position, the trajectory of the club head, the plan of thetrajectory of the club head, the trajectory of the ball, a plane, adirection, angle or the like of the ball, the strike position, angles ofimpact of the club head and ball may be illustrated, as may any otherinteresting parameters.

In the situation with the golfer striking or being about to strike agolf ball, the distance from the golfer's shoes to the ball may be ofinterest as may the vertical and horizontal distance from the golfer'shead to the ball. Such information may be derived from the image andillustrated in the image as lines, rulers, numbers or the like.

In one situation, the controlling means are adapted to adapt an image inrelation to radar data obtained at least substantially simultaneouslywith the image. This has the advantage that the altering of the image isperformed so that, e.g., real time parameters may be seen. If a golferstriking a golf ball is imaged in a video sequence, the path/trajectoryof the club and/or ball may be determined on the basis of the radar dataand visualized in the image/video.

Alternatively, the image and radar data may be stored, and the image(s)altered subsequently with radar data from a plurality of points in timeor a time period. Normally, the point in time of providing the image isincluded in the points in time or time period of the providing of theradar data. Thus, an image or a video sequence may be adapted or alteredby illustrating therein a trajectory, position, velocity, accelerationor any relation thereof of one or more moving elements imaged in theimage/video. Thus, the full trajectory, for example, of a ball or clubmay be illustrated in all images of a video sequence of a swing or animage relating to a point in time in relation to the swing.

Preferably, the radar and imaging device are fixed in relation to eachother. In this manner, a field of view of the radar may overlap with afield of view of the imaging device. Thus, elements viewed by theimaging device may be sensed by the radar, and the radar data may thenbe introduced in the image. Naturally, the fields of view need not beidentical, but a certain overlap is preferred, as this overlap may bethe primary working area/angle of the assembly.

In a preferred embodiment, the assembly further comprises means forderiving position data relating to a flying object imaged by the imagingdevice and positioned in a field of view of the radar. In thissituation, the object will be in the above primary working area/angle ofthe assembly.

In this situation, deriving means may be adapted to derive the positiondata from both the radar data and the image. In general, data relatingfrom radars may have a relatively low angular precision or certainty,whereas that of an imaging device may be rather high. Contrary to that,a distance between the imaging device and an imaged object, or arelative velocity may be difficult to determine when using standardcameras. Thus, angular measurements may be obtained from the imagingdevice, whereas distance/velocity data may be determined on the basis ofthe radar data. Combining such data, the position acceleration, velocityor the like of an object may be determined with a certainty higher thanthat obtained by the imaging device and the radar alone.

In one embodiment, the controlling means is adapted to determine one ormore positional relationships between elements identifiable oridentified in the image. Such relationships may be provided in the imageas e.g. positions, distances, angles or the like. The identification ofelements in images is an old and well known technology.

Another aspect of the invention relates to an assembly including a radarand an imaging device both adapted to provide information relating toone or more objects. The assembly further includes a controller adaptedto receive radar data from the radar and an image from the imagingdevice and adapt the image in relation to the radar data. The assemblyfurther includes a processing element adapted to receive an image fromthe imaging device, identify an element therein, and generate an alteredimage in which the element seems rotated.

In this respect, the imaging device and radar may be as those of theabove aspect. Also all elements of the individual aspects of theinvention may by interchanged, as the skilled person will understand.

The rotation is a well-known technique, and may be performed in a numberof manners depending on the image(s) available as well as the computingpower available. If only a single image is available, the rotation maybe obtained using the so-called perspective transformation. If more thanone image is available, such as when stereo imaging (the providing oftwo images from different angles and at the same point in time) is used,the rotation may be performed by generating an image by combining thetwo initial images. In this situation, the individual pixels or pointsin the resulting image are determined on the basis of the correspondingparts of the initial images. In this situation, different weights may beput on the individual initial parts of the initial images depending onthe distance from the actual part of the scene imaged to the individualimaging device. The larger the distance, the lower the weight the partof the pertaining image is given.

If enough cameras are used, any angle may be selected, and it ispossible to view e.g. a person from any angle desired. This is standard3D vision technology.

In one situation, it is desired to rotate the scene in a manner so thata pre-determined angle, direction, axis or set-up is obtained. This maybe achieved by the processing element being adapted to determine, fromthe radar data and/or the image, a predetermined direction or axis, andto provide the altered image wherein the direction or axis is rotatedinto a predetermined direction or axis. A suitable direction or axiswill depend heavily on the imaged scene and the individual situation.Natural directions or axes may be present in some scenes, such as roads,railway tracks, horizontal or vertical lines or the like.

In other situations two or more positions or elements may be identified(in the image or by the radar), and directions or axes be defined therefrom.

In a particular situation where a golfer is aiming to launch a golf ballfrom a tee position to a target position, which may be identified by aflag, the direction from the tee to the target may be selected as thedirection. In this situation, the tee position may be determined fromthe image/video data at a point of time before launch, or from the radardata at or after launch (see WO 2006/002640) and the target position maybe identified in the image either by pattern recognition or by a personindicating the target in the image (see e.g. WO 2006/002639).

Having determined the position/direction/axis, the rotation of the imagemay be performed.

It should be noted that in addition to the rotation, the image may alsobe scaled (such as cropped), in order to standardize the contents of theimage. In addition, the image may be adapted to have a standard angularsection and perspective.

This scaling or other adaptation may be in order to have apre-determined element have a predetermined extent in the image, or adistance have a predetermined extent in the image.

This may be advantageous in a situation where the assembly furthercomprises means for storing the altered image and providing this at alater point in time for comparison with a second, altered or not, image.Thus, having altered the image into the desired set-up, this may nowmore easily be compared to other images rotated/scaled to the sameset-up.

In the situation of the golfer, the images may be rotated/moved to allhave an axis between tee and target in the same direction and beingscaled so that all golfers have the same height. Then, the postures,angles (club handle to arm, upper/lower arm, legs, between directions ofmovement ball/club or the like) and relative positions (golf ball toshoe, head in relation to golf ball or the like) may be compareddirectly.

In a particularly interesting embodiment, the processing element isadapted to, in the altered image, provide data relating to the radardata, the relating data provided being rotated correspondingly.

In that situation, the radar data and other data may also be rotatedand/or scaled in order to describe the rotated scene. Especially whenthis data is to physically illustrate lines, curves, distances,positions in relation to elements imaged in the image, thisrepositioning is desired.

Then, this added data may still describe the scene of the image in thesame manner as before rotation.

Naturally, the assembly may further comprise means for storing therotated data and provide this at a later point in time for comparisonwith second, rotated data.

Then, when comparing the rotated/scaled image to another image, thisadditional data may also be compared, whereby additional information ispresent and an in-depth comparison may be made.

Thus, pros or amateurs may compare their swing, performance, distances,looks, postures etc. with others, such as trainers, pros or champions—orthemselves in order to monitor progress.

In a preferred embodiment, the assembly further comprises a secondimaging device, the second imaging device being calibrated in relationto the imaging device and/or the radar.

In this respect, the calibration typically will be to the effect thatthe viewing direction or field of view of the second imaging device isknown in relation to that/those of the other imaging device and/orradar. In this manner, e.g. stereo vision is possible, as is thecorrelation of elements imaged by the second imaging device and sensedby the radar.

Naturally, this calibration may be performed in a number of manners,such as using visible elements imaged by both imaging devices and/orvisible by the second imaging device and sensed by the radar.

Naturally, this second imaging device may be fixed in relation to theother imaging device and/or the radar, which makes the calibration easy.Alternatively, it may be freely movable in relation thereto, which mayrequire a re-calibration when the relative positions are changed.

In general, the controlling means may be adapted to receive an imagefrom the second imaging device and adapt the image thereof in relationto the radar data. This adaptation may be as that of the other image,whereby, again, the positions/trajectories/planes/directions etc. may beintroduced, as may other information relevant to the scene imaged.Preferably, the image from the second imaging device is adapted inaccordance with the calibration, so that if this image is provided fromanother position/angle/distance, the image is adapted in accordancetherewith. Thus, e.g., a direction or axis is introduced correctly inrelation to the different viewing angle of the second imaging device.

In general, the radar data or video images may additionally be used forestimating a velocity of an imaged or sensed element and to adapt areplay speed (frames per second) on the basis of this velocity. From aseries of frames, a velocity of an element moving during filming may beestimated by the displacement of the element between frames. Using aradar, such movement may be estimated virtually in real time.

Replaying the video, the frame rate may be reduced, when the velocity ofthe imaged/sensed element exceeds a threshold, such as to betterillustrate the movement. Alternatively, the frame rate may constantly beadapted to the velocity in any desired manner.

An aspect of the invention relates to a system including a radar and animaging device both adapted to provide information relating to one ormore objects. The assembly further includes a controller adapted toreceive radar data from the radar and an image from the imaging deviceand adapt the image in relation to the radar data. The system furtherincludes one or more radar reflecting elements positioned inpredetermined positions in relation to the assembly. The radar isadapted to provide information relating to a relative position,distance/direction and/or angle/rotation between the reflectingelement(s) and the assembly, and the controller is adapted to adapt animage received from the imaging device in accordance with the relativeposition/angle/rotation/direction/distance. Again, the radar and imagingdevice may be as those of the above aspects, as may the calibrationsetc.

In this respect, a radar reflecting element may be any elementdetectable by the radar. A number of radars can only detect movingobjects, whereby the reflecting element may comprise a moving part, suchas a fan or the like, which is detectable by the radar. An alternativeis the simulation of a moving object by providing a reflecting elementhaving a transponder outputting a signal emulating a moving object.

In general, and also for radars able to detect immovable objects, it maybe desired to ensure sufficient reflection of the radiation. This is astandard concern.

In the present aspect, the reflecting elements are positioned inpredetermined, normally fixed or stationary, positions in relation tothe assembly. Preferably, such positions are within the field of view ofthe radar, or the assembly may be brought to a position where theelement(s) is/are within this field of view. Naturally, a number of suchelements may be used, of which not all are in the field of view at thesame time.

The radar is adapted to provide information relating to a relativeposition/distance and/or angle/rotation between the reflectingelement(s) and the assembly. Then, the controlling means are adapted toadapt an image received from the imaging device in accordance with therelative position/angle/distance/rotation.

If the controlling means has access to information relating to thedistance, positions or relative distances or positions of the elements,the controlling means may be adapted to determine a position of theassembly in relation to the elements and thereby the surroundings. Inthis manner, the initial positioning of the assembly may require lessstringency, as the assembly is adapted to itself determine itsposition/rotation/distance. Then, this information may be used foradapting the image.

This adaptation of the image may simply be the introduction ofinformation relating to the rotation/position/distance of the assemblyin relation to the surroundings, such as the introduction of the cornersof the world, pointers toward interesting places or objects, distancesto certain elements or the like.

An especially interesting embodiment is one further comprising means forreceiving information relating to a position or area in the surroundingsof the assembly, the controlling means being adapted to provide, in theimage, information relating to the position or area. This informationmay be derived by the assembly or may be received from anotherelement/sensor or the like. Thus, this knowledge may be used foroverlaying the image with information relating to objects ordemarcations in the surroundings, such as the lines of a soccer field, ahole position of a golf course or the like. In addition, equidistantlines, squares or elements may be provided in order to determinerelative distances between elements in the surroundings. Naturally, suchelements may be determined or identified from the radar data and/or theimage.

An interesting situation is where the position is not a particularelement in the image but simply a position, such as an estimated landingzone or point of a flying object, such as a golf ball or a baseball. Inthis situation, this position may be illustrated in the image, and infact the precision of the estimation may be illustrated as a circlesurrounding (such as rotated to the ground plane of the surroundings)the estimated landing spot. As the landing spot estimation becomes moreprecise, this circle may be reduced in size.

In one situation, the assembly is rotatable/movable in relation to thesurroundings, and the controlling means is adapted to estimate arotation/movement or rotational position from relative position(s) ofthe reflecting element(s). Thus, even if the assembly moves, thismovement may be determined from the resulting displacements of thepositions of the reflecting elements in relation to the radar.

In that situation, the controlling means may be adapted to highlight theposition or a part of the area which lies in the image, such as whenthis is in the field of view. When the assembly rotates, this isdetermined by the radar, and as the image as a result of the rotationchanges, a re-determination of the position/area is made so as to keepindicating the correct position/area in relation to the imaged scene.Naturally, if the image is zoomed, this may be taken into account.

In this manner, the imaging device of the assembly may be used fortracking a golf ball in flight, while tracking, using the radar, theactual direction of the assembly and while receiving informationrelating to the landing zone. When the landing zone (or part thereof)enters the field of vision of the imaging device and therefore is seenin the image, the landing zone information may be provided, as may ahighlighting of the hole, if desired.

Alternatively, the assembly may overview or pan the landing zone areaand “await” the touchdown of the ball which a part of the time is notimaged and not sensed by the radar, while the illustration of theestimated landing point is illustrated.

Also, during flight, a pointer toward e.g. the hole, a distance, aheight of the ball, a velocity or any other interesting information maybe provided in the altered image.

Another aspect of the invention relates to a method of operating anassembly including a radar and an imaging device. The method includesproviding information relating to a plurality of objects, the radaroutputting radar data, and the imaging device outputting an image. Themethod further includes determining, from the radar data, movementand/or position data, of objects imaged by the imaging device andpositioned in a field of view of the radar. The data describespositions, directions, trajectories or planes of movement of theobjects. The method further includes providing movement data to the dataat a correct position in the image. The movement data illustrates thepositions/trajectories/planes/directions of the objects in the image.The determining step includes identifying, from the radar data and/orthe image, a position of impact of a first object of the objects and asecond object of the objects. The providing step includes illustrating,for the first object, a spin after the impact and, for the secondobject, a 3D impact vector at the impact.

In this respect, the radar and imaging device may be as those describedabove.

Again, the adaptation may be that of introducing relevant information inthe image, the relevant information being derived e.g. from the radardata.

In one situation, the determining step comprises identifying, from theradar data and/or the image, a position of impact of two of the objects,the providing step comprising illustrating:

for a first of the objects at least one of: a trajectory, a 3D launchvector and spin after impact and

for a second of the objects at least one of trajectory, direction/angleof movement and 3D impact vector at impact.

Then, the determining step could comprise providing information relatingto an impact between a golf club and a golf ball, the golf ball beingthe first of the objects and the golf club being the second of theobjects.

As described above, different positions, such as that of a tee, a head,a foot or the like, may be illustrated, as may trajectories of movingobjects, distances between elements, angles of elements, planes, strikepositions or the like. Thus images or video may be provided with thisadded information describing the movements imaged.

The determining step preferably comprises determining one or more of,and the providing step comprises illustrating one or more of:

a face angle of the club at impact,

a relation between the ground and an axis of a club head of the club atimpact, and

a point of impact of the club head.

In one situation, the radar data are obtained at least substantiallysimultaneously with the image. In other situations, image data may beadapted using radar data from a plurality of points in time or a periodof time. As mentioned above, the point in time of taking an imagepreferably is included in the points in time or time period of providingthe radar data.

Preferably, the method has the initial step of fixing the radar andimaging device in relation to each other, preferably so that a field ofview of the radar overlaps with a field of view of the imaging device.In this manner, a calibration of the fields of views etc. need beperformed only once.

In one embodiment, the method further comprises the step of derivingposition data relating to a flying object imaged by the imaging deviceand positioned in a field of view of the radar, the deriving stepcomprising deriving the position data from both the radar data and theimage. Thus, angle data may be used from the image and distance/velocitydata may be used from the radar.

Preferably, the adapting step comprises providing, in the image, datarelating to the radar data. Normally, this data would relate to anelement imaged in the image, but this is not a requirement.

In a particular embodiment, the method further comprises the step ofdetermining one or more positional relationships between elementsidentifiable or identified in the image. In the above golfer situation,the distance between the ball and the feet or vertically or horizontallyto the head may be interesting.

In this situation, the positions may be derived from the radar and/orthe image.

Such positions and positional relationships (distances, angles,projections or the like) may be provided in the adapted image.

In another aspect, the invention relates to a method of operating anassembly including a radar and an imaging device. The method includesproviding information relating to one or more objects, the radaroutputting radar data, and the imaging device outputting an image. Themethod further includes adapting the image in relation to the radardata, and determining, from the radar data and/or the image, apredetermined direction or axis. The step of providing the altered imageincludes providing an altered image wherein the direction or axis isrotated into a predetermined direction or axis.

As mentioned above, a number of techniques exist, depending on theavailability of processing power, the number of imaging devices etc.

Preferably, the method further comprises determining, from the radardata and/or the image, a predetermined direction or axis, and whereinthe step of providing the altered image comprises providing an alteredimage wherein the direction or axis is rotated into a predetermineddirection or axis. As mentioned above, this may be followed or precededwith a scaling of the image. The direction/axis or positions used forgenerating or identifying these may be derived both from the radar dataand/or the image.

In this situation, preferably the method further comprises storing thealtered image and providing this at a later point in time for comparisonwith a second, altered or not, image. Having provided the desiredrotation and possible scaling, comparison is facilitated.

Naturally, it may be desired that the step of providing the alteredimage comprises providing, in the altered image, data relating to theradar data, the relating data provided being rotated correspondingly, sothat this additional data is maintained or also provided. Thus, thisdata is, if required, rotated/scaled in the same manner.

In this situation, it is preferred that the method further comprises thestep of storing the rotated data and providing this at a later point intime for comparison with second, rotated data. In this manner, not onlythe images but also the radar data may be compared.

In a preferred embodiment, the method further comprises the step of asecond imaging device generating a second image, the second imagingdevice preferably being calibrated in relation to the imaging deviceand/or the radar. This calibration may be performed by imaging visibleelements also imaged by the other imaging device and deriving acorrespondence between the positions in the two images of the element.Alternatively or in addition, an element may be imaged by the secondimaging device and sensed by the radar in order to provide acorrespondence between positions in the image and in relation to theradar.

Also the calibration may be a timing relationship ensuring that thetiming relationship between the second imaging device and the radarand/or the other imaging device is known.

One embodiment further comprises the step of the second imaging devicegenerating an image and the step of adapting the image in relation tothe radar data. This adaptation may be as that mentioned above. If thesecond imaging device is positioned in another position than the otherimaging device, the adaptation preferably is different, taking intoaccount the different viewing angles of the images.

As mentioned above, the radar data and/or video images may also be usedfor estimating a velocity of an imaged and maybe sensed element, thisvelocity may be used for controlling the frame rate when replaying thevideo.

An aspect of the invention relates to a method of operating an assemblyincluding a radar and an imaging device both adapted to provideinformation relating to one or more objects. The assembly includes acontroller adapted to receive radar data from the radar and an imagefrom the imaging device and adapt the image in relation to the radardata in an environment comprising one or more radar reflecting elementspositioned in predetermined positions in relation to the assembly. Themethod includes providing information, via the radar, relating to arelative direction/distance, position and/or angle/rotation between thereflecting element(s) and the assembly, and adapting an image receivedfrom the imaging device in accordance with the relative direction/distance/position/angle/rotation.

In this respect, when the reflecting elements are stationary in thesurroundings, the direction of the assembly, as determined from therelative positions of the assembly and the reflecting elements, may beintroduced in the image.

If the assembly is immovable, the relative positions need be determinedonly once by the angles between the radar (and thus assembly) and thereflecting elements. If the assembly moves, these relative positionswill move.

Naturally, the adaptation of the image may be any adaptation relating tothe relative positioning or movement.

One situation further comprises the step of receiving informationrelating to a position or area in the surroundings of the assembly andproviding, in the image, information relating to the position or area.This information may be provided on the basis of the radar/imagingdevice or may be received from another element or sensor. This positionor area need not be present in the image, but if so, it may beidentified as described above.

A preferred embodiment is one further comprising the step of rotatingthe assembly in relation to the surroundings, and estimating a rotationor rotational position from relative position(s) of the reflectingelement(s). In this situation, information relating to therotation/movement may be introduced in the image.

Alternatively or additionally, the adapting step could comprisehighlighting the position or area in the image. As described above, thisposition may enter or exit the field of view, and if the assembly moves,the position in the image of identifying this position/area will change,in order for it to be identified correctly in relation to thesurroundings.

Even though the above description has been exemplified with a golferexample, the same functionality may be obtained in any other situationin which imaging and motion detection/determination is desired, such asin other sports (baseball, tennis, table tennis, cricket, soccer,football, handball, volley ball, basketball, or the like), as well as inother situations where a movement is desired imaged andquantified/illustrated at the same time.

As mentioned above, the individual aspects may be combined so that theassemblies of the first and second aspects may be one and the same, andthe assembly used in the third of final aspects may be those of thefirst and second aspects, for example.

BRIEF DESCRIPTION OF THE DRAWINGS

In the following, the invention will be described with reference to thedrawing wherein:

FIG. 1 illustrates an assembly according to the invention imaging agolfer,

FIG. 2 illustrates an image or video provided of the golfer with radardata provided,

FIG. 3 illustrates a close-up image or video of a club at the time ofimpact, and

FIG. 4 illustrates a rotatable assembly.

DETAILED DESCRIPTION OF EMBODIMENTS

In FIG. 1, an assembly 10 is illustrated having a radar 20 and a camera30 both mounted within a housing 12. The radar 20 and camera 30 arecalibrated to each other so that position data obtained from a movingobject and using the radar 20 may be converted into a position in animage or video provided by the camera 30. This calibration may beprovided using one or more elements being visible, so that the camera 30may image it, while being adapted to reflect radar radiation so that itsposition may be determined by the radar 20,

The radar 20 and camera 30 are provided so as to have at least partlyoverlapping fields of view, so that movements of one or more objects inthese fields of views may be both imaged and determined/quantified bythe radar.

In FIG. 1, the golfer 50 swings a club 52, so as to swing it along aclub trajectory 54, and hits a ball 56 which subsequently flies along aball trajectory 58.

The movements of the club 52 and ball 56 are determined by the radar 20while the camera 30 images the movements in one or more images, such asa video sequence with a predetermined number of frames per second.

From the data from the radar 20, it is possible to determine positions,velocities, angles and the like from the different moving elements (club52 and ball 56) at different points in time, and this data may becorrelated with images provided by the camera 30 at the same points intime. In this manner, both image data andmovement/position/velocity/acceleration data may be provided.

In addition, from the image(s), see FIG. 2, additional data may beprovided, such as distances or relative positions of elements in animage. Interesting parameters for a golfer 50 is a distance between thegolfer 50 and the ball 56 before striking the ball 56, such as from thegolfer's shoes to the ball 56. Another interesting relative position isthe position of the golfers head in relation to the ball 56, along thehorizontal plane.

In order to determine such data, it is desired to know the distance fromthe camera 30 to the golfer 50 or ball 56. This distance may bedetermined in a number of manners, such as from the apparent size of theball 56. As all golf balls have a standard size, other dimensions of theimage may be derived.

Another manner is to determine the distance from ball 56 to radar 20from the tee position 60 as determined subsequent to launching the ball56. This is described in WO 2006/002640.

From the radar data, the trajectory of the club 52 head may bedetermined and may be provided in an image of the golfer 50 taken duringswinging of the club 52. In addition, the trajectory of the ball 56,such as a 3D ball launch vector and ball spin, may be provided in theimage data, and the direction/angle 64 of movement of the club 52 head,such as 3D club impact vector 66, at impact may be compared to thedirection of the ball trajectory 58 after impact (see FIG. 3). Fromthese directions, a face angle of the club 52, i.e. the angle of theclub surface actually impacting the ball, may be determined, which is ofinterest to the golfer 50.

Actually, the trajectories of the club and ball may alternatively oradditionally be determined from the video, if desired.

Another interesting parameter is the so-called lie angle which relatesto the relation between the ground and the axis of the club head(whether the club has the heel or the toe closest to the ground). Thismay be determined from the video or image.

Actually, it may be desired to provide a grid or the like at the groundor from the ground up at the golfer in order for the golfer to know andcompare (see below) the relative positions of the feet, tee 60, head andthe like.

Also, the camera 30 may be programmed or set to provide a close-up imageor video of the ball 56 and club 52 at launch (see FIG. 3). A close-upvideo may be taken with an increased number of frames per second inorder to gain a better insight of the parameters at launch. In additionthe video or images may be provided with the determined trajectories54/58 and/or directions etc. in order to illustrate the relativemovements at or around launch.

From such images, the actual point of impact of the club head may bedetermined which is also of interest to the golfer.

In FIG. 3, a club 52 about to strike a ball 56 is illustrated along withdata relating to the club trajectory 54 and the ball trajectory 58. Fromthis data, the impact location of the ball on the club face can bedetermined. This is done by determining the location of the ball in theimage and then subsequently comparing the location of the club head inan image taken at point of impact. If an image is not available atexactly the time of impact, the club trajectory data determined by theradar can be used for determining where the club head will be movingfrom the location in the image just prior to impact or prior to an imagejust after impact.

It is noted, as described above, that a velocity, for example of theball, determined using the images or the radar, may be subsequentlycontrolling the frame rate when replaying the video. Thus, the framerate may be reduced when the velocity increases in order to betterillustrate the video in spite of the increased velocity.

Reverting to FIG. 2, and as is mentioned in WO 2006/002639, a golfermay, in the image, identify a target 62, which is aimed at. Then, as thetee or point of launch 60 and the target 62 are identified, a coordinatesystem may be set up with origo in the tee 60 and an axis along thedirection from the tee 60 to the target 62. This is the main coordinatesystem of the golfer 50.

However, the positioning of the assembly 10 in relation to the golferscoordinate system will depend on a number of parameters, whereby it isfar from certain that image data obtained at different points in time(whereby the assembly may be positioned at different relative positions,distances, angles) may easily be compared. In order to providecomparable data, the coordinate system is determined in the originallyobtained image/video/radar data, where after the image(s) is/arerotated, using simple image manipulation such as perspectivetransformation, so as to have the axis from tee 60 to target 62 be alonga predetermined direction in the altered image. In addition, a scaling(enlargement/reduction) or translation of the image may be desired inorder to have the distance from the tee 60 to the assembly 10 be asdesired, or other image manipulation such as to obtain a desired angularsection or perspective.

This rotation may also be performed of the radar data, so that the sametrajectories/positions or the like may be illustrated correctly also inthe rotated/scaled image. This is a simple operation, as therotation/scaling is known.

After this altering of the image(s), comparison is possible. Then, thesame golfer 50 may, independently of the relative position of theassembly, store a number of swings for later comparison, or he/she maycompare a swing to that of another golfer, such as a trainer, a pro orthe like. Also, the trajectories, planes, angles and the like may becompared rather easily.

Naturally, if more than one camera 30 is provided, the rotated image maybe generated on the basis of multiple images (preferably provided ortaken at least substantially the same point in time), in any of a numberof manners.

In one situation, the generation of the image from multiple images maybe performed by identifying, for each part or pixel in the final image,corresponding parts of the initial images and weighting the informationin the two initial images on the basis of the distance from thepertaining imaging device to the object. In this manner, the imagingdevice with the smallest distance will be given the largest weight forthat part of the final image.

If a second camera 32 is provided, this should be calibrated to thefirst camera 30 and/or the radar 20 so that a correlation of elements inthe image of the second camera 32 with elements imaged by the firstcamera 30 and/or as detected by the radar 20 is possible.

This calibration may also be a correspondence between generation ofimages with the image generation of the first camera 30 and/or theobtaining of data by the radar 20. A number of manners of obtaining thisare known such as the forwarding of instructions to the second camera 32as to when to generate an image, the providing with the images from thesecond camera 32 of timing information describing a point in time ofderiving the image, or the like, so that images or image and radarinformation corresponding in time (such as derived within apredetermined maximum time period) are selected and used.

In this manner, it may be possible to actually provide, in theimage/video of the second camera 32, data relating to the radar data inthe same manner as described above for the image/video of the firstcamera 30.

Naturally, the information derived may be presented in a number ofmanners, such as an image having therein trajectory data (ball and/orclub), target/tee positions and wherein images of the golfer atdifferent points in time during the swing may be overlapped.

FIG. 4 illustrates another manner of using the assembly 10, which is inconnection with one or more radar reflecting elements 70 positioned inthe surroundings of the assembly 10 and within (at least at times) thefield of view of the radar 20.

A radar reflecting element 70 may be an element reflecting the radiowaves either by providing a moving element (such as a fan or flag) or byreceiving the wave and outputting a corresponding wave (so-calledtransponder).

When the assembly 10 is rotatable and rotating, the position of areflecting element 70 will move within the field of view of the radar20, which will then determine this movement or position. Knowing therelative positions of the elements 70 will, even duringmovement/rotation of the assembly 10, make the assembly 10 able todetermine its position and/or rotation/rotational position, in relationto the surroundings from the positions determined by the radar 20.

In this situation, the images provided by the camera 30 of the assemblymay be adapted to this movement/rotation/position/rotational position.

In one situation, an element in the surroundings has a known position inrelation to the assembly 10 or the element(s) 70 even if not imaged bythe camera 30 and/or determined/detected by the radar 20. This positionmay be known from measurements of another assembly 10 or thoseillustrated in the above-mentioned WO-references.

Nevertheless, it may be known or estimated that this element will enterthe field of view of the camera 30, whereby the image of the camera 30may be altered to reflect this. One manner of reflecting this may be toillustrate where or when in the image the element will enter.

However, it is clear that when the assembly 10 moves, the point of entryof the element into the image of the camera 30 will change, but due tothe constant monitoring of the movement/rotation/etc. of the assembly 10by the radar 20 and potentially a constant or intermittent updating ofthe position/uncertainty, this change may be determined and the imagethereafter be made to reflect the new position upon movement of theassembly.

In a particularly interesting situation, the camera 30 may image anexpected or calculated/estimated touch down spot or landing spot of astruck golf ball or baseball. This landing spot may be determined in anydesired manner such as from a radar as that described in WO 2005/116678.

From e.g. a determined trajectory of the flying golf ball or baseball,the actual position and landing spot may be constantly determined inorder to provide an estimate of the landing spot with greater andgreater accuracy.

In the image of the assembly 10 imaging the estimated landing spot, theestimated landing zone may be illustrated by a circle having a diameterreflecting the uncertainty of the landing spot determination. As theball approaches the landing spot, the certainty will increase and thecircle illustrated in the image of the camera 30 may be madecorrespondingly smaller. Also, the landing spot or center of the circlemay move.

Naturally, other types of data may be illustrated, such as an estimatedtrajectory of the ball, an expected rolling of the ball or the like.

This illustrating, due to the operation of the radar 20 and the elements70, is no longer dependent on the camera 30 and the assembly 10 beingfixed in relation to the surroundings. If the assembly 10 rotates, forexample, the rotation will be detected by the radar 20, as theposition(s) of the element(s) 70 will change. From this change, therotation may be quantified, and the landing spot or other position ofthe surroundings may be correspondingly altered.

Then, the camera 30 may be rotated to follow the path of the ball 56 inflight while illustrating, when the estimated landing spot enters thefield of view of the camera 30, the landing spot estimate or otherdesired data.

Naturally, a zooming in or out of the camera 30 will alter the relativepositions of the element(s) 70 in the image. As the zooming is known,however, this is easily determined so that zooming in may be performedwith the corresponding amendment of the added data, such as landing spotposition and uncertainty.

Even though the figures and pertaining description have focused on theuse of the present assembly for use in golf, the same functionality maybe obtained in any other situation in which imaging and motiondetection/determination is desired, such as in other sports (baseball,tennis, table tennis, cricket, soccer, football, handball, volley ball,basketball, or the like), as well as in other situations where amovement is desired imaged and quantified/illustrated at the same time.

What is claimed is:
 1. A system for illustrating the flight of a sportsball, the system comprising: a radar; a first imager configured to imagea moving sports ball; and a controller receiving radar data from theradar and an image from the first imager and altering an output image ofthe ball in flight based at least in part on the radar data toillustrate a portion of a trajectory of the moving sports ball in theoutput image.
 2. The system of claim 1, wherein the output image isgenerated by a second imager.
 3. The system of claim 1, wherein trackingthe sports ball as it moves through a common field of view of the firstimager and the radar simultaneously using both the first imager and theradar.
 4. The system of claim 2, wherein the second imager provides aseries of output images and the controller alters the series of outputimages to illustrate the portion of the trajectory.
 5. The system ofclaim 1, wherein the controller is configured to output the alteredoutput image while the moving sports ball is in flight.
 6. The system ofclaim 1, wherein the controller is further configured to: project theportion of the trajectory onto a predetermined plane; and illustrate theprojected portion of the trajectory in the altered output image.
 7. Thesystem of claim 4, wherein, in at least one of the series of outputimages, the illustrated portion of the trajectory ends at a position inthe at least one output image at which the moving sports ball is locatedin the at least one output image.
 8. The system of claim 1, wherein theimager and the radar are fixed relative to one another.
 9. The system ofclaim 1, wherein the controller is further configured to illustrate atarget in the altered output image.
 10. The system of claim 9, whereinthe controller is further configured to illustrate a direction from alaunch position of the moving sports ball to the target in the alteredimage.
 11. The system of claim 1, wherein the controller is furtherconfigured to illustrate a 3D launch vector of the moving sports ball inthe altered output image.
 12. The system of claim 1, wherein thecontroller is configured to illustrate, in the altered output image,numeric values associated with movement of the moving sports ball, thevalues including at least one of a launch speed, a launch angle, alaunch spin rate, an apex height, a carry distance and/or a curvature.13. A method for illustrating the flight of a sports ball, comprising:receiving, from a radar, radar data associated with the sports ball;receiving, from a first imager, at least one image including the sportsball; and altering an output image including the sports ball in flightbased at least in part on the radar data to illustrate a portion of atrajectory of the moving sports ball in the altered output image. 14.The method of claim 13, further comprising: outputting a series ofoutput images including the sports ball in flight, and illustrating theportion of the trajectory of the sports ball in the series of outputimages.
 15. The method of claim 14, wherein the series of output imagesis generated by a second imager.
 16. The method of claim 14, wherein, inat least one of the output images, the illustrated portion of thetrajectory ends at a position at which the sports ball is at a point oftime of generation of the at least one output image.
 17. The method ofclaim 13, further comprising: outputting the altered output image whilethe sports ball is in flight.
 18. The method of claim 13, furthercomprising: projecting the portion of the trajectory onto apredetermined plane; and illustrating the projected portion of thetrajectory in the output image.
 19. The method of claim 18, wherein thepredetermined plane is a horizontal plane of a coordinate system havingan origin at a point from where the sports ball was launched.
 20. Themethod of claim 13, further comprising: illustrating the portion of thetrajectory at positions of the sports ball in the output image.
 21. Themethod of claim 13, further comprising: illustrating a target in theoutput image.
 22. The method of claim 21, further comprisingillustrating a direction from a launch position of the sports ballprojectile to the target in the output image.
 23. The method of claim13, further comprising illustrating a 3D launch vector of the sportsball in the output image.
 24. The method of claim 13, furthercomprising: tracking the sports ball as it moves through a common fieldof view of the first imager and the radar simultaneously using both thefirst imager and the radar.